Method for Controlling Electronic Devices Using Digital 

Recall Tool 

Reference to Related Application 

5 

This application is a continuation-in-part of U.S. patent application serial 
u no. 09/880,397, filed June 12, 2001, which is a continuation-in-part of U.S. patent 

q application serial no. 09/785,049, filed Feb. 15, 2001, for which priority is claimed. 

m 

u ] The entireties of the prior applications are incorporated herein by reference. 
1 10 

Background of the Invention 

: ;:2 i 
],£ 

m The copending patent application captioned above describes a graphical 

: s:;s? 

fu user interface for a machine having a screen display. A significant feature of the 
15 interface is that it provides the means for hand drawn entry of on-screen objects 
which may be associated with functions, files, connections, and other objects or 
actions accessible by the system to carry out the user's desired purpose, whatever 
it may be. A fundamental aspect of this interface is the ability to carry out 
commands and actions in real time corresponding to the hand drawn inputs as 
20 they are applied to any on-screen object, including (but not limited to) objects 
representing files (text, data, sound, video, graphics, photos, and the like), 
functional devices (audio processing, video processing, graphic and photo 
processing, text editing, data processing, internet communications, test equipment, 
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medical devices, and the like), and interconnecting arrows and lines that link 
these files and functional devices in on-screen arrangements that are drawn by 
the user to accomplish desired tasks. The prior related applications also introduce 
arrow logics as a method for inputting transactions involving objects displayed 
5 onscreen through the use of arrows and lines drawn between such objects. 

It is noted that approximately 26% of the population of the US and 
: = approximately 5% of the world population (including the US figure) own a PC 

a 

□ today. These statistics show that although current times are widely known as the 
y computer age, ownership and, to a significant extent, the use of computers is the 
%4 10 exception rather than the rule, in much of the world, even the developed "first" 
world. 

M- One possible reason for the reluctance of some to embrace the PC is a 

m difficulty for them to feel comfortable or familiar with many of the PC's required 
Nl operations. In other words, the PC requires a certain learning process associated 
15 with its operating system ("OS") and third party software that run on this OS. 
The operation of a given OS is determined by the manufacturer of the OS and, like 
it or not, users must comply with the means, methods, and rules which the OS 
requires to be followed, even if this operation is not intuitive for the users. 

This same model applies to virtually every computer software program that 
20 runs on a PC, from graphics and video software to text software to business 

software. Each of these software programs carries with it an instruction manual 
that inexorably increases in size and complexity with each new software update. 
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Pull Down Menus 

Typically, to operate a computer, i.e., a PC or Mac orits equivalent ("PC"), 
pull down menus, task bars, icons, and the like are provided by the software 
application and required to be used. Pull down menus have proven to be a 
5 deterrent to a myriad of would-be computer users. Pull down menus started out 
nearly two decades ago as simple lists of functions. But today, as more and more 
u functionality has been added to computers, pull down menus have become 
g complex worlds unto themselves. In fact, many a user has thrown up his hands in 
|;JJ frustration because a certain software program, with which these users had 
:;10 developed great proficiency and familiarity, can no longer be easily operated, 
J " because the pull down menus were changed in a new software update. Such 
P changes could include eliminating familiar items, new locations of menus entries, 
m new categories where controls, features, and tools are grouped in a new way, the 
fj changing of names of menu items, etc. To many users this phenomenon creates 
15 difficulty because "nothing is familiar anymore". Therefore the very familiarity 
that supported their proficiency is removed, all in the name of progress called a 
software update. 
The Look of a Software Screen 

Another important factor in maintaining a user's mastery and familiarity of 
20 certain OS or other software programs is the "look" of the screen when 
operating this software. The "look" can be more than an appearance of 
graphics. This "look" can affect in a greater or lesser degree the very operation 
of the software itself. Things as simple as font changes (new font sizes, styles, 
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etc.) or different graphic layout schemes, or in some cases, changes in color 
schemes can all have negative affects on both the ability of a new user to adapt 
to or develop a proficiency in any given software or the ability of experienced 
users of certain software to continue their proficiency beyond and into one new 
5 software update after another, where the very "look" that was relied upon as a 
familiar and efficient environment is changed time and again in software updates. 
j»* Thus there are two general issues cited here affecting users of software: 

□ 1) New users may not embrace, adapt to, or accept a certain software or OS 

UJ because its "look and feel" and its operation is not easily understood or 

10 assimilated by them. 

y i 

:L 2 ) Existing users (including experienced users) may find their proficiency 

; : as? 

£ so compromised by changes in the software from one update to another that their 

% ability to "keep up" is compromised and the usefulness to them of certain 

■ y software becomes diminished. 
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Summary of the Invention 



The invention generally comprises a graphic tool, termed herein the DRT 
(Digital Recall Tool) that enables a user to draw a simple diagonal line to establish 
at least three things: 

a) Where an element is to be placed; 

b) How big the element is to be, namely its width and height; 

c) What is the definition of element; that is, is it a layout, a control panel, a 
piece of text, a software tool, a series of operation devices, i.e., faders, knobs, 
joysticks, switches, etc. 

In addition, contexts or zones can be applied to the DRT to further define 
what the element will be when a diagonal of a certain color is drawn within a 
particular context or zone. 

The DRT may have multiple assignments made to it. These assignments 
can be as simple as a single graphic object or a single word in text, to very 
complex graphic or device control panels or layouts or complex presentation 
documents, which may include pictures, animation, sounds that playback and can 
be edited or mixed, or hand drawn graphics, or typed text or even video. 

With a single stroke of a pen or mouse or any other input device, the DRT 
can be used to place onscreen anything that has been assigned to it. The length 
of the stroke determines the width or length of each object that makes up the 
layout, controls, presentation, etc. ("elements"), that is being recalled by the DRT. 
The slope of the stroke determines the height or vertical size of the same. 
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If the DRT stroke equals a distance that is less than the original width of 
the element that is being recalled, the element is automatically resized 
proportionately (each individual graphic object, text, and/or device making up the 
element is proportionately resized). If the DRT stroke equals a distance that is 
5 less than the height of the element that is being recalled, the height of each 
individual item comprising the element is proportionately resized to a smaller 
height. 

□ There are various ways that the drawing of a simple DRT line can produce 

; jJ (recall, place onscreen) an assigned element (after DRT is turned on and a line 
40 specifier is selected): 

; 1 . A user draws a line and on the mouse up click or when the pen is lifted from 
the display, the element appears automatically resized proportionately 

;0 according to the length and slope of the line drawn. 

- 2. As a user draws a line on a display, the element "flows" out of the pen or 

15 mouse stroke. What this means is that "as the stroke occurs", the element is 
revealed as though its items are flowing out of the mouse cursor or pen in real 
time. In this case, the drawer of this line does not see a line at all. What is seen 
is the revealing of ever increasing proportional resizing of the element until 
the mouse or pen stops drawing. 

20 3. The user draws a line and on the mouse up click or when the pen is lifted from 
the display the specifier appears onscreen. The specifier is clicked or touched, 
and the element assigned to it appears onscreen. 
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4. The user selects "non-proportional" in the DRT info window. The user 
selects a line specifier, and then draws a line onscreen. On the up click or pen 
lift off the element assigned to the selected line specifier appears onscreen in 
its full original size, so that the size of the recalled element is not affected by 
5 the size of the line drawn onscreen. 

Note: if under scenario 2 above, the pen or mouse does not initiate a 
u mouse up click or its equivalent, the element can be immediately resized by 
q continuing to drag the pen or mouse down, up, left or right. In this case, the 
hj element will appear to proportionately stretch to change the proportionate size of 
\J0 each item that makes up the element. Once the pen is lifted from the display or 

the mouse up click occurs, the element appears onscreen in a form that cannot be 
j«* altered by drawing a DRT line. In fact, drawing a second DRT line may produce 
m another copy of the element, depending upon which DRT color is selected. The 
fU size of this second element, of course, will be determined by the length and slope 
15 of the second drawn line. 

Note: If the DRT isturned "off, drawing a line will not recall an element 
stored in the DRT tool. Rather, merely a line will be drawn on the screen. 
Engaging the DRT function. 

One way to engage the function of this invention is to draw a graphic that 
20 a software system recognizes as an input command for a DRT switch. Then this 
DRT switch is touched to turn it "on" and to activate the DRT function. 
Touching this switch again will turn it "off and deactivate the DRT function. 
An example of a drawn DRT command is drawing a rectangle inside of a 
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rectangle. The drawing of this object would result in the automatic recognition 
and creation of a DRT switch, which in turn could be touched (turned on) to 
activate the DRT function.. 

Likewise, a verbal command can be used to engage the DRT function. 
5 Such a spoken command could be "DRT" or "Digital Recall Tool", or any 
designated nickname for this function, like, "Slash", etc. 
U Advantages of the DRT 

0 Some of the advantages of the DRT over the prior art are: 

UJ Typical PC pull down menus can be reduced or eliminated. The DRT can be 

40 enlisted to access any element, including software tools, images, files, functions, 
^ formats, views, tables, windows, devices, layouts, software applications, etc., that 

are typically accessed by various pull down menus, icons, etc., in a PC 
2 environment. 

1 u Single Stroke Recall The DRT enables a user to draw a single stroke anywhere 
15 onscreen, which could take less than 1 second, to access anything that is stored in 

the DRT. This single stroke can also be used to determine the height and width 
(outer dimensions) of whatever is being recalled. In other words, the longer the 
stroke, the wider the element being recalled; the steeper the slope, the greater the 
height of the element being recalled. So, users can not only determine what they 
20 recall, but also the size and proportion of what they recall, as well as its location. 
Again, this can all be determined with a single stroke onscreen. 
Entire control panels or layouts can be assigned to a DRT. Fully functioning 
controls and operational devices or systems, i.e., an oscilloscope, a video playback 
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device, animation or automation controls, an entire audio mixing console or video 
editing system, medical apparatus, etc., can be placed anywhere onscreen, with 
any proportional size, with a single stroke of a pen or mouse. 
User-Defined Controls Through Drawing. Users can design their own controls, 
layouts, presentations, web pages, letters, brochures, etc. ("elements"), and then 
assign any number of these "elements" to a DRT. By recalling these "elements" 
at will, users can control the "look", functionality and operability of their 
onscreen environment With single DRT strokes of a pen or mouse, users can 
utilize the DRT to quickly recall any number of user-defined "elements". By this 
method users can completely control the simplicity or complexity of their 
onscreen world. 

Control of "Element" Placement and Its Size Through Drawing. The DRT can 
be used to create a space of any height and width that can determine the width of 
text that can be typed. This DRT "space" will cause text to be automatically 
wrapped when it "hits" the right side of this DRT space. Thus text blocks of any 
width can be setup with the stroke of a pen or mouse - drawing a diagonal with 
the DRT function "on". This approach can be used, e.g., to quickly setup up 
columns of text for magazine or brochure layouts. 

Drawing DRT lines to Place Pictures and Video. The DRT can be used to place 
pictures onscreen in and around text. The simple drawing of a DRT stroke 
onscreen can place a picture and at the same time determine its height and width. 
The same approach can be used to place a video viewing screen anywhere on a 
display. 
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DRT "Zones". The DRT can be programmed to recall a completely different 
element when a DRT is drawn in different "zones" on a display screen. In other 
words, the DRT can be programmed to recognize different areas of a display as 
being separate and distinct sub-sections, each sub-section having a respective set 
5 of definitions of its DRT elements. When the same DRT stroke is drawn in each 
of these sub-section areas, the element that is recalled can be completely different. 

J;* DRT Auto Sequencing. The DRT can be programmed to automatically sequence 

; ^ the recalling of elements when a same colored DRT line is drawn in the same 
"zone". For example, drawing six blue DRT lines in the same "zone" could 

3(0 yield six different elements being recalled sequentially and placed in this zone. 

% The six elements would be automatically sequenced, each next element appearing 

; 7 with each next blue DRT line drawn in the same "zone". 

:*:s=: 

% Software Upgrades Can't Create an Unfamiliar Environment Since users can 
use the DRT to customize their onscreen world and since users can determine 

15 what is drawn and where they draw it and what this recalls, software updates 
cannot place controls in unwanted or unfamiliar locations. Users determine the 
placement of controls, tools, text, pictures, etc., by drawing onscreen. These same 
users determine what colors are assigned to what elements. If "zones" are being 
used, users determine the size and shape of these "zones" and how they are 

20 going to be used. In short, users are controlling their onscreen environment by 
what they draw, NOT by what a manufacturer puts into a menu or task bar, or its 
equivalent. 
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Brief Description of the Drawing 



Figures 1A-1B are a sequence of screen views showing an input drawn to 
evoke the DRT tool, and the resulting DRT switch used to turn the DRT function 
on and off. 

Figures 2A-2F depict the drawing of a diagonal DRT line and the resulting 
DRT element that is scaled according to the outer dimensions of the drawn line. 

Figures 3A and 3B are a sequence of screen views showing an input 
drawn to evoke the onscreen inkwell for arrow logic functions. 

Figure 4 is a screen view depicting a possible DRT Info Window. 

Figure 5 is a screen view showing one technique for assigning elements to 
a DRT color selection. 

Figure 6 is a screen view showing the assigned elements of Figure 5 
recognized by the DRT system. 

Figure 7 is a pop-up window displayed to name a DRT assignment 
following the steps of Figures 5 and 6. 



Patent Application 

Method for Controlling Electronic Device Using Digital Recall Tool 

Page 11 



Figures 8A and 8B are a sequence of screen views showing, respectively, a 
technique for removing a color from the DRT inkwell, and the resulting 
rearranged color array. 

Figures 9A and 9B are a sequence of screen views showing, respectively, a 
technique for adding a color to the DRT inkwell, and the resulting rearranged 
color array. 

Figure 10 is a screen view showing an arrow logic used to assign a text 
portion to a DRT color selection. 

Figure 1 1 is a screen view showing a list of color assignments for the DRT 

tool. 

Figure 12 is a screen view showing a technique for displaying the list of 
assignments of elements to a DRT color selection. 

Figure 13 is a screen view showing the list of DRT colors and their 
respective assignments. 

Figure 14 is a screen view showing the DRT Info Window displayed with 
the Zones ON switch activated. 
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Figure 15 is a screen view showing a technique for drawing a plurality of 
zones on the screen display. 

Figure 16 is a screen view showing the zones resulting from the steps 
5 shown in Figure 15 being assigned to a DRT color. 

u Figure 17 is a screen view showing the DRT list of assignments to the color 

q blue, with Zones switched ON. 

JO Figure 18A and 18B are screen views showing the DRT list of assignments 

« to the color blue, with Zones switched ON and Sequencing ON, and techniques 

h* for making assignments for sequencing and zones. 



U Figure 19 is a screen view showing the zones portrayed previously and 

15 DRT elements being placed in each zone. 

Figure 20 is a screen view showing the results of the DRT placements of 
Figure 19. 

20 Figure 21 is a block diagram showing a plurality of computers connected 

to a network for sharing DRT elements, such as a fader switch array. 
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Figure 22 is a block diagram showing a plurality of computers connected 
to a network for sharing DRT elements, in this example a text element. 

Figure 23 is a block diagram showing a plurality of computers connected 
to a network for sharing DRT elements, in this example a three channel audio 
processor. 

Figure 24 is a depiction of a palette for displaying line specifiers, showing 
various examples of line specifiers. 

Figure 25 is a depiction of a DRT Info Window corresponding to the line 
specifiers of Figure 24. 

Figures 26A-26C are three views of different methods for displaying a line 
specifier as the DRT line is executed. 
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Description of the Preferred Embodiment 



The invention generally comprises a graphic tool that is intended to be 
used in conjunction with a computer or any electronic device that employs a 
5 screen display as an interface with the operator of the device. (In this application, 
the term "screen display" is used to denote any graphic that is visible on a 
computer display screen or its equivalent.) The graphic tool, termed herein the 

□ 

□ DRT (Digital Recall Tool), enables a user to draw a simple DRT line to call forth 
Ly graphical elements that represent useful objects. 

10 In this application, the following terms may be found, and are defined 

« thusly: 

H line specifier (assignable specifier) - a visible characteristic, such as line 

CO style, color, texture, symbol, and alphanumeric character, which may be used to 
!"U distinguish DRT lines and represent one or more elements which have been 
15 assigned to a line specifier. 

assignment - the association of an onscreen element with a DRT line that 
is identified by its line specifier. 

element - an onscreen object (i.e., email, console, picture, text file, video 
clip, web page link, etc.) that can be assigned to a DRT line. 
20 occurrence - the drawing of a DRT line to recall the element assigned to 

the DRT line. 

outer dimension - the overall height and width that are defined by the 
drawing of a single line. The length (which could be an average length or 
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derived length) and angle or slope (which could be an average slope or derived 
slope) of such drawn line determines an "outer dimension". This outer dimension 
is not necessarily a visible graphic onscreen. But it can determine the overall size 
of any element that is recalled by the drawing of a DRT line. 
5 Another way to derive the outer dimensions for an assigned element may 

be to select a menu entry that defines the size of the element to be recalled. Of 
course, this menu entry may be user-alterable. 
^ electronic device - a computer, which includes laptops, desktops, hand 

held PDAs, cell phones and the like, and which also includes a computer logged 
JO onto the Internet. Electronic device also includes any electrical appliance that 

incorporates a microprocessor and display means. 
\2 functional object - a graphic that includes, at least in part, a functional 

no device that can be operated by any means, including the manipulation of the 
fU graphic or associated graphics, verbal input, written input, typed input or any 
15 other input to a computer. Such functional object can be operated as almost any 
type of equipment, action, operation, cause, etc. Such functional object could 
include, but is not limited to any of the following: a switch, knob, joystick, fader, 
an audio player or editor, a video editor or player, an oscilloscope, a piece of 
medical equipment, an email,a web link, a network, a presentation, an animation, a 
20 sequence of events, etc. 

behavior - indicates a graphic or its equivalent that includes at least one 
instruction, condition or action or its equivalent that can be applied to anything 
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associated with that graphic. For instance, a rectangular perimeter could be a 

graphic, or it could have a behavior, such as functioning as a zone. 

To enable the computer system to recognize a simple DRT line as a 

command to display the useful object(s), the system includes at least one method 
5 to invoke the DRT function. With regard to Figure 1 A, the user may draw a 

graphic that represents a DRT switch, such as a rectangle nested (drawn within) a 
u larger rectangle. The system recognized this particular drawn graphic as a DRT 
□ switch command, and responds by displaying the DRT switch shown in Figure 
uj IB. Thereafter, touching (or mouse clicking, or the equivalent action) the DRT 
10 switch will activate the DRT function, and touching it again will turn off the DRT 

function. 

k Likewise, a verbal command may be used to engage the DRT function. 

£0 Such a spoken command may be "DRT" or "Digital Recall Tool," or any 

rU designated nickname for this function, such as "Slash" or the like. 
15 Whenever the DRT function is activated, the user may draw a DRT line, 

using a mouse, touch screen, pen, finger, or any other input device that enables 
drawing inputs. As shown in Figure 2A, a diagonal line 21 is drawn from its 
origin 22 to any desired length and at any angle or slope to the tip 23 of the line. 
The horizontal width W and the vertical height H of the line 21 define outer 

20 dimensions which determine the overall size of any element recalled to the screen 
display, or its equivalent, by the drawing of this diagonal line. 

The type of element recalled by the drawing of a line onscreen may be set 
by the user or defined in other ways. For example, the DRT tool may be set to 
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display an array of slider devices (faders), all of which may be set to be activated 
individually or collectively or in any combination. As shown in Figures 2B and 
2C, the array of faders is scaled in size to fit the outer dimensions 24 of the drawn 
line - the size of these outer dimensions is set by the height and width of the 
5 diagonal line 21. Note that the size and spacing of the faders are scaled in 

proportion to the outer dimensions defined by the diagonal line 21. Furthermore, 
u it should be noted that drawing the diagonal with a greater or lesser slope (which 
□ can in many cases be determined by the origin and end points of the line) would 
i y result in the fader array being displayed as wider or narrower and that all faders 
40 are displayed, regardless of the outer dimensions that are created by the diagonal 

line. Further note that an outline showing the perimeter of these outer 
■!** dimensions is not necessarily visible to the user of this invention. In fact, for 
ffl many applications it will be invisible and only the resulting size of the element 
= y (i.e., a row of faders as depicted in figures 2B and 2C), assigned to a line's 
15 specifier, may be visible. 

With regard to Figure 2D, an actual hand drawn DRT line may comprise a 
somewhat erratic or irregular line 21'. The software detects the pen down origin 
22' and the pen up end point 23', and sets the size of any element assigned to 
any assignable specifier selected as a line style. The size of this element is 
20 determined by measuring the length and slope of the line created between the 
two points 22' and 23'. 

With regard to Figure 2E, another example of a drawn DRT line determines 
the size of one or more elements assigned to that line's specifier. Even though 
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line 21" is irregular, it still defines the outer dimensions for any and all elements 
that are assigned to its specifier. Such outer dimensions are determined by this 
length and slope of this line. The origin 22" and end point 23" are used to 
calculate the outer dimensions defined by line 21", irrespective of the 
5 irregularities of the line 21". Furthermore, line 21" is shown recalling an array of 
knobs 24". This element has been previously assigned to a line specifier which is 
u first selected as a line style and then drawn as a DRT line. The size of this graphic 

i sss l 

□ element 24" is determined by the length and slope of the diagonal line 21". 
Ly In the example of Figure 2F, the recalled element comprises an ellipse 24' ' ' 

|0 having a width and height that is calculated from the length and slope of the 
drawn line 21" ' 

H Note that the outer dimensions may be defined by any regular or irregular 

CO DRT line, as in Figures 2D-2F, not just a straight diagonal line; in many cases, the 
rU outer dimensions are portrayed with invisible borders, so that the only item that 
15 appears is the element that has been assigned and recalled and sized to conform 
to the outer dimensions. 

A fundamental and powerful aspect of the DRT tool is that the user may 
assign a wide number and variety of elements to line specifiers that are selectable 
within the tool. These elements may include a function, action, operation, layout, 
20 control panel, series of devices as in a audio or video mixer/editor or test 

equipment, one or more hand drawn graphics, one or more pictures, one or more 
videos, any type of text (including letters, brochures, emails, books), one or more 
data files, and the like. Each element may be assigned to a different specifier in 
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the DRT system, and any element may be selectively recalled by the user drawing 
a single stroke on the display in a line style, which could be a color, object, 
texture, type of geometric line (e.g., a dash, dotted or slotted line) that has been 
associated with the element that the user wishes to call forth. 
5 To assign an element to a color in the DRT system, the user first initiates an 

arrow logic, or its equivalent, as follows. With regard to Figure 3A, the user may 
H* draw a graphic, such as a rectangle and a horizontal line therewithin, which is 
□ recognized as a command to display an onscreen arrow logic color/line palette, 
W hereinafter termed "onscreen inkwell." As a result, the system provides the 
iO onscreen inkwell display of Figure 3B, in which a variety of color choices and 
s ;_ line width choices are displayed. The user then selects a color that is associated 
[ '" A with an arrow logic that has a transaction defined to take the objects from which 
; ;5 the arrow or line is drawn and place those objects in the object to which the 
1 y arrow or line is drawn. In this example, the desired arrow logic color is green, and 
15 the selection of this color causes the inkwell to be outlined in green to confirm 

the selection. For a more detailed description of arrow logics, refer to the pending 
patent application serial no. 09/880,397, filed June 12, 2001. 

Thereafter the user may click or touch the DRT switch to activate the DRT 
feature. As shown in Figure 4, the DRT switch object then turns to green (or any 
20 other designated color). When the user right clicks (or double clicks, or double 
touches, or invokes the equivalent as a vocal command) on the activated DRT 
switch, the system responds by displaying an Info Window, as shown in Figure 4. 
The Info Window presents a list of functions that may be selected to be activated. 
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The user selects "DRT Inkwell", which elicits a display, shown in Figure 5, that 
comprises a color palette (including a List switch) that comprises the DRT 
Inkwell. 

The user may then draw an arrow to select any onscreen object or plurality 
5 of objects, and direct these objects to one color presented by the DRT Inkwell, as 
shown in Figure 5. The system responds by displaying a white arrowhead at the 
head end of the arrow, as shown in Figure 6. The white arrow head indicates that 
the software recognizes the intended assignment to a color. When this arrowhead 
|:5f is clicked or touched, the software displays a pop-up window, as shown in Figure 

■liar 

: JIO 7, that prompts the user to enter a name for the DRT assignment that is being 
'U made. After a name is entered and the OK button is clicked, the system stores the 
y : assignment of the fader array to the color blue. 

pi Thereafter, whenever the user activates the DRT system and selects the 

blue color and draws a line, the system will display the fader array that was 
15 assigned to the color blue, in proportion to the length and slope of the DRT line 
that is drawn. 

Note: the difference between the onscreen inkwell and the DRT inkwell is 
that the onscreen inkwell is used to select a color for drawing for any non-DRT 
purpose onscreen. Whereas, the DRT inkwell is used to permit users to assign 
20 elements to individual colors and then recall those elements by drawing diagonal 
lines or their equivalent with a color selected from the DRT inkwell. 

Another simple method for selecting and assigning an element to a DRT 
color is to first turn on the DRT system and call forth the DRT inkwell, then touch 
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(or click) one of the DRT colors, and then touch (or click) the onscreen object 
that is to be assigned to the selected color. The system will recognize these 
inputs as a DRT assignment command, and will then display a pop-up window for 
naming the assignment, as shown previously in Figure 7. 
5 The user may remove any color from the DRT inkwell. To remove a color 

from the DRT inkwell, the user right clicks (or its equivalent mechanical action or 
U vocal command) on any color and selects "Delete" in its Info Window, or drags 
the color out of the inkwell into blank onscreen space and perform a mouse up 
W click or a pen lift off, as shown in Figure 8A. Upon this action, this color will be 
iO deleted from the DRT inkwell and will disappear from onscreen, and the inkwell 
will reconfigure itself to be contiguous, as shown in Figure 8B. 

To add a color to a DRT inkwell, the user clicks or touches and holds on a 
color rectangle in the onscreen inkwell (not the DRT inkwell), as shown in Figure 
1 y 9A. This will duplicate the color rectangle. Then the color rectangle (in this 
15 example, magenta) is dragged so it overlaps one of the color squares on the DRT 
inkwell. As this onscreen inkwell color rectangle is dragged over the DRT 
inkwell's color squares, a colored vertical line 26 indicates the point in the DRT 
inkwell where the onscreen color rectangle will be inserted between two existing 
color squares. In addition, the shape of the onscreen color rectangle will 
20 automatically change to match the shape of the DRT color squares when the pen 
is lifted from the screen, or upon the mouse up click, as shown in Figure 9B. 

With regard to Figure 10, a further assignment in the DRT system may 
involve using an arrow logic to assign a letterhead to the color light yellow. This 
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assignment is labeled in the pop-up window as "letterhead for Leonard 
Whitman". 

It may be appreciated that with the DRT system, the line color is only one 
way to assign an onscreen element to the DRT tool. In a more general sense, any 
convenient form of assignable line specifier may be used as a line property to 
which an element may be assigned. With regard to Figure 24, the DRT system 
may be provided with a palette 51 of assignable line specifiers that do not involve 
color, although they may be combined with color to further multiply the 
assignment options. For example, row 52 of the palette 51 displays examples of 
line styles that vary from dashed to picket to gradient to solid line. Row 53 
displays examples of various line textures, row 54 displays examples of various 
identifiable objects, and row 56 portrays letters (or numbers). Any of these 
palette items may have one or more onscreen elements assigned to it, using the 
same methods described herein with regard to the DRT color palette. 

The DRT Info Window for assignments, as shown in Figure 25, is similar to 
the Info Window of Figure 1 1, in that it equates each line specifier with its 
respective assignments. The process of implementing the DRT line specifiers of 
Figures 24 and 25 is very similar to the method described above. First, the user 
creates a DRT switch, and turns on the system. By double clicking or right 
clicking on the DRT switch, the DRT inkwell or palette 51 of Figure 24 may be 
called forth and the user would then select an assignable line specifier from this 
inkwell. Thereafter, a line drawn will call forth the assigned element(s) assigned 
to this line specifier. For example, with regard to Figure 26A, assuming that a 
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three channel fader array has been assigned to the line specifier "A", the user 
draws the line 57. When the pen up (or mouse up) signal is received as the line 
57 is completed (a DRT occurrence), the assigned element: the three channel 
fader array, appears onscreen. Its outer dimensions are determined by the length 
5 and slope of the drawn line. 

Likewise, a similar realization of the DRT, shown in Figure 26B, involves 
; ai the same steps leading up to drawing the line 57, except that there is no dashed 
□ letter displayed as the line 57 is drawn. When the line is completed (a DRT 
Ly occurrence), the assigned line specifier" A" appears, and comprises an active 
tO switch element onscreen. Upon touching or clicking the "A", it disappears and is 

immediately replaced by an element assigned to the "A" specifier.. 
! si The line specifier "A" may be transmitted over a network or its equivalent 

ffi to any other electronic device, where it will appear as an "A", and this character 
rU on the display of the recipient electronic device likewise may be activated to call 
15 forth the elements assigned to it. That is, the transmitted specifier also conveys 
the elements assigned to it. 

Referring again to Fig. 26B, the "A" character may be considered to be 
the element, as the faders are in Figures 26A, C, and D. This "A" may be an 
object having other objects or devices assigned to it. Thus the drawing of a line 
20 would call forth onscreen the "A" (or any object having one or more 

assignments to it), where the "A" is the onscreen element. The difference here is 
that this onscreen element (A) has devices, actions, functions, etc. assigned to it 
which can be placed onscreen by touching or clicking on the "A". Thus this 
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"A" element could be sent via a network and the germane material (assigned to 
"A") can remain hidden until the user of the recipient device decides to access 
that material. Various methods for assigning items to graphic objects are 
described in copending application serial no. 09/880,397, filed June 12, 2001, 
5 entitled Arrow Logic System for Creating And Operating Control Systems. 

Another format for displaying the DRT, shown in Figure 26C, also involves 
. L drawing the line 57. As the line is drawn, it is not displayed onscreen. Rather, the 
5 assigned element (the fader array) first appears as a small figure at the origin of the 
Q line, and then grows and expands as the line progresses, as suggested by the 
JjO sequential size increase that is shown. Upon pen up or mouse up, the fader array 

is displayed at a size determined by the length and slope of the line 57. 
u With regard to Figure 26D, the user may select a line specifier, as before, 

m and also select "Proportional OFF". Thereafter, when the user draws a DRT line 
rU of any length to call forth the assigned element, the element will appear at the 
15 same size as the original element. This size is not affected by the length or slope 
of the DRT line. 

The display format of Figure 2 may also be employed with these or any 
other line specifiers. Other formats for displaying this transaction may be 
apparent to those skilled in the art. 
20 With reference to Figure 4, the Info Window includes a toggle switch area 

labeled "Proportional ON/' indicating that any element recalled by the DRT will 
be displayed completely displayed and proportionally sized according to the 
length and width of a DRT line or its equivalent. However, there may be 
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circumstances in which it is preferred not to control the proportional size of an 
assigned element by the outer dimensions defined by a drawn DRT line or its 
equivalent. For example, when recalling a piece of text using the DRT tool, such 
as a paragraph from a text file, it is often desired to display the text in its original 
5 font size, margin parameters, line spacing, and the like. To generate a full size 
display, the user may first click or touch on the "Proportional ON" entry to 

; u toggle it to an OFF state. Thereafter, all DRT elements recalled are full size, unless 

o 

□ and until the Proportional switch is once again toggled to the ON state. Note that 
Ly when an element is recalled with Proportional OFF, the length and height of the 
■10 diagonal drawn to call forth the DRT element does not determine the size of the 
display of the recalled element. Rather, the element appears in full size; that is, the 

.sat 

M same size as it was when it was assigned to DRT. Either the origin point or end 
point of the DRT line may set a boundary of the elements to be portrayed. The 

• r U location of such boundary may determine the location of the upper left corner or 
15 lower right corner, etc. of the elements being displayed onscreen. The advantage 
of this feature is that the user may draw a short quick line to recall an element of 
any size, even one that fills the entire screen. The disadvantage of this feature is 
that the length and slope of the line cannot be used to resize or determine the size 
of the element being recalled. 

20 At any time when the DRT tool is turned on, the user may click or touch 

the "List" or "L" rectangle of the DRT inkwell, as shown for example in Figures 
8, to access a list of the DRT colors and their respective assignments. As shown 
in Figure 11, this list will indicate, for the examples illustrated herein, that the color 
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blue has had 7 faders assigned to it, and light yellow has had the letterhead of 
Leonard Whitman assigned to it. Note that the heading "Show Color Names" is 
a switch that may be touched to toggle on and off the display of the names of the 
respective colors, for example to aid those individuals with limited color vision. 
5 Also, this List Info Window may be used to select any color appearing therein to 
be the designated DRT color for a subsequent DRT rectangle. In this regard, it 
has the same function as the DRT inkwell. 
□ To further extend the scope of the DRT tool, provision is made to assign 

! In 

Ly multiple elements to one color, although generally only one element would be 
10 recalled at one time. With regard to Figure 12, when the DRT inkwell or palette is 

m 

displayed, any color may be right clicked or double clicked on , after which the 
Ms color rectangle is highlighted and the system calls forth a display of a list of 

elements assigned to that color. In the example of Figure 12, the blue assignments 
* y include the aforementioned letterhead, three different jpeg pictures, and a text 
15 footer. The element that is active (the one that will be displayed when blue line is 
drawn with the DRT tool active) is the element that is highlighted; in this 
example, the letterhead for "Leonard Whitman." All the items in the list are 
operable; that is, any one may be clicked and selected to be the active element. 
Thus a large number of elements may be stored and easily recalled by the DRT 
20 tool. 

In the Info Window of Figure 1 1 , the user may click on any one color 
rectangle to expand the window to display the assignments made to the selected 
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color, as shown in Figure 13. Note the scroll slider 28 that enables access to a 

long list of colors in a small Info Window. 

Another important aspect of the DRT tool is the ability to provide zones 

on the display, which comprise delimited areas of the display, each area having its 
5 own unique set of DRT recognition conditions, available tools, and/or rules of 

operation. As shown in Figure 4, after turning on the DRT system, the DRT Info 
M, Window includes an entry labeled "Zones OFF." This entry may be clicked or 
□ touched to turn On the Zones function, as shown in Figure 14. Thereafter, a color 
Ly or other specifier that has no zone assigned to it, such as blue in this example, is 
10 clicked on and thus selected.(Note that the color blue does have multiple 

elements assigned to it, but it does not have any zones assigned to it.) Then one 
M> or more diagonal lines are drawn (in any direction) across a selected portion of 
M the display, as shown in Figure 15, defining (in this example) three distinct zones 
i'U 31-33. Note that the transitory diagonal lines are dashed blue, indicating the 
15 designation of zones, and that each zone is outlined by a rectangle whose size is 

determined by the diagonal that forms it. Also, the DRT switch continues to 

"float" above the zones display. 

NOTE: If "Zones" has been turned on (in the DRT Info Window or in a 

DRT inkwell color's Info Window) then the zone assignment equals a behavior. 
20 If "Zones" is off then the assignment of the three boxes created by the three 

diagonal lines is in accordance with the user-defined assignments, as shown for 

example in Figure 12. So, "Zones on" enables assigning "zones" to a DRT 

color, which zones will act as a behavior. With "Zones off, assigning "zones" 
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to a DRT color will represent objects that will be recalled by drawing a DRT line 
onscreen. NOTE: if a color is selected as a line style where such color has no 
assignments of any elements made to it, a possible default operation (which is 
being referred to just above) is that drawing a line with a color that has no 
5 elements assigned to it will recall a window or its equivalent. This may be 
designated by a visible outline. 

As a behavior, "zones" will enable a user to program a same colored 
□ diagonal line to be drawn in different zones and be used to recall different 
S *U elements. In other words, a single colored DRT line can be used to recall 

" |0 completely different elements when this diagonal is drawn in different zones. To 

j.fj 

^ set up this type of operation requires having "Zones on" in the Info Window of 

the color for the DRT inkwell or in the Info Window for the DRT switch itself. 
j:5f Shown in Figure 16 are the three zones 31-33 that are being assigned as 

1 * zones to the color blue in the DRT inkwell. To do this, right click on the DRT 
15 switch or its equivalent to see its Info Window. In its Info Window select "DRT 
inkwell". If this Inkwell has been previously selected (turned "on"), then it will 
already be visible onscreen. To assign these three zones (as zones and not as 
objects) to the color blue for the DRT, the following steps are taken: (1) Make 
sure that "Zones" is turned on in the Info Window for the color blue (Figure 17). 
20 (2) Select an arrow color (for example, dark yellow) that conveys the logic 
transaction: "place every item that is visible onscreen and place these item(s) 
inside (assign them to) the object that the arrow is pointing to." (This arrow color 
is selected in the onscreen inkwell, not the DRT inkwell, (3) Draw a dark yellow 
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arrow so that it points to the blue square in the DRT inkwell. With the mouse up 
click or the pen lift off, the arrow head turns white. Touch the white arrow head 
to confirm the transaction, and a dialog box will be displayed, and the zone 
combination can be named. Upon the mouse up click or pen off from touching 
5 the white arrow head, all three "zones" get stored in the color blue for the DRT. 
Finally, upon the up click or pen off, the three zones can, as a user option, 

u disappear from onscreen, leaving the DRT and its inkwell visible. 

n NOTE: If the DRT switch changes color (changes the color of a perimeter 

[j rectangle appearing around the DRT switch) to indicated the currently selected 

Q 

40 color in the DRT inkwell, the user may make an assignment to this color by 

drawing a line/arrow so that the arrow head or end of the line overlaps or points 
M- to the DRT switch. 

B There are many methods to assign individual elements to a single color, 

m using zones, in a DRT Inkwell (or Palette). In one method, the following steps 
15 may be taken: 1) Right click on a color in a DRT inkwell for which multiple 

elements have been assigned. This will call forth the Info Window for this color; 

2) Turn on "Zones" in this Info Window for this color. In the case of this 

example, it's the color blue, (3) Assign "elements" that have been assigned to a 

given color (in this case, blue) to specific zones. 
20 There are many ways to assign specific elements that were stored for the 

color blue to the zones that have been assigned to the color blue. Below are 

three of them: 
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1) Click to view zones in the Info Window for the color blue. When the 
zones appear, click and drag the element from the blue Info Window into the 
zone where you want a drawn blue DRT line to cause this element to appear. 
This action, of course, requires the user to remember where each zone is located 
5 onscreen, or to make the zones visible onscreen. One way to visualize each zone 
is to show an outline around the perimeter of the zone. 
:u 2) Make the element assignments to zones directly on the blue DRT 

□ inkwell Info Window. There are many ways to do this. As one example, type or 
Uj hand print the word "Zone 1" or just "1" next to or on top of the blue element 
10 that you wish to see appear when you draw a blue diagonal in zone 1. This will 
have the same result as what is described under #1 above, as long as "Zones" is 
u " on " i n the Info Window for the color blue. 

CO 3) Perform the same steps as described under item 2 above, except first 

i'U touch or click the desired element to select it and then say the name of the zone, 
15 e.g., "Zone 1" or "1", instead of typing or printing it. To assign other blue 

elements to other zones, type or say the name of the other zone, e.g., "Zone 2" or 
"2" for another element, and then "Zone 3", etc. 

With regard to Figure 18 A, there is shown an example of typing "Zone 1" 
on the DRT Info Window for the color blue to assign an element (the letterhead) 
20 to Zone 1. Also illustrated is the typing of Zone 3 on the footer, and zones 2a-2c 
on the jpeg elements. Thus, it may be appreciated that when a user draws a blue 
diagonal line in Zone 1, the letter head for Leonard H. Whitman will appear. If, 
however, a blue diagonal line is drawn in Zone 3, a footer "Confidential 
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Information to Leonard H. Whitman" will appear. One way of knowing which 
zone is being drawn in is to use the point of the first pen down or mouse click. 
Whichever zone contains this point is the zone that determines which element 
will be recalled with a drawn diagonal line. 

A further, powerful aspect of the DRT system is that multiple elements can 
be sequenced for a single colored DRT line for a single zone, so that repeated use 
of a DRT diagonal line of a particular color causes a series of assignments to be 
made in a preset order. Continuing in Figure 18B with the example of Figure 18 A, 
the user may wish to draw more than one blue DRT line in more than one location 
in Zone 2. In other words, when the first blue diagonal is drawn in Zone 2 one 
type of picture appears (whose height and width are governed by the length and 
angle of the drawn blue diagonal line). Then when a second blue diagonal line is 
drawn in Zone 2, a different picture appears, and so on. This is an example of 
auto-sequencing for elements assigned to a single color for a single zone. 

To implement auto-sequencing of elements for a single color in a single 
zone, (consider the three different pictures for Zone 2 for the color blue), the 
following steps are taken: (1) Turn "Sequencing" on in the Info Window for the 
color blue for the DRT, (2) then Type Zone 2 over or near the first element that is 
to be sequenced in Zone 2. Then with the Sequencing feature still "on" in the 
blue Info Window, touch the next elements that you want to appear with each 
new blue diagonal line drawn in Zone 2. As each element is touched is will get a 
sub-number to indicate its order of sequence selection, (3) draw three blue 
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diagonal lines in Zone 2 and you will see the three different pictures that were 
sequenced to Zone 2 appear. 

An alternate approach to the sequencing of the three pictures would be to 
drag each picture to Zone 2 in the order that you want them to appear when you 
5 draw multiple blue diagonal lines in Zone 2. 

Another alternate approach would be to draw an ellipse around a number 
l A of elements (in this case pictures) that were assigned to a specific DRT color or 
specifier. Then type or print a sequence of numbers or letters or a combination of 
j both, i.e., 1 - 3 or 1 , 3, 2 or A - C or B, A, C, or 1 A to 1C, etc., close to or 
CIO overlapping the selected elements. This technique, illustrated in Figure 18B, will 

sequence these elements for the color they are assigned to. Note: if only one 
CI element has been assigned to a zone, even if sequencing is on, only that element 
□3 will be recalled. 

ry Note: If the user wishes to see a graphic outline of the zones (i.e., as 

15 shown in Figure 16) that have been assigned to the color blue as a behavior, the 
following steps are taken: Click on the "Show Zones". When this is selected, a 
graphic outline of each assigned "behavior" zone will appear onscreen. This is a 
necessity for users who wish to assign elements to zones by dragging duplicates 
of these elements into a zone of their choice. It would be difficult to drag to 

20 zones that are not visibly defined. 

Note: When an element is dragged from a DRT color or specifier Info 
Window to a zone, this element is automatically duplicated and is generally not 
removed from the Info Window. A duplicate is dragged for programming 
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purposes. On the mouse up click, this duplicate disappears from the Zone to 
which it was dragged and the text "Zone 1, Zone 2, Zone 3, Zone 4", etc., 
appears next to the element that was just duplicated and dragged to a particular 
zone. 

5 With regard to Figures 19 and 20, the result of the setting up of three 

zones for the color blue as described above is three zones 31-33, although they 
u need not be visible to be functional and, in fact, users may opt not to have these 
jSj zones be visible. If the user is creating a letter, first select the color blue in the 
hj DRT inkwell. Then draw a blue DRT line in Zone 1. This places the letterhead for 
r|0 the letter. If it is desired to place the letter head in the middle between the page 
margins, draw the diagonal in the middle of the zone, or to the left or right for 
placing the letter head in those locations. Five blue diagonals have been drawn, a 
rfi task that requires about 5 seconds or less, and the resulting effect, shown in 
ry Figure 20, is quite significant. In fact, the use of DRT lines with zones eliminates a 
15 significant amount of accessing of menus, selecting of options, resizing of objects 
(if this resizing is even possible in an existing software program), setting up 
margins, etc, that are required in state of the art computer applications. All of 
these functions, which are necessary on every PC, are eliminated with the DRT 
and its zones feature. 
20 Note: Continuing to type text or add graphics to the zone, where the 

vertical height of that zone is exceeded, many results are possible. One possible 
result is that a scroll bar or its equivalent can appear for that zone. In this latter 
case, a zone can be extended into pages of text, rather than being limited to one 
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page whose text is scrolled through. And in this latter case, a scroll bar will 
appear to enable scrolling through pages, as in prior art word processing 
programs. 

With regard to Figure 21, a further aspect of the invention is the ability of 
5 more than one computer to share DRT tool elements, transactions, and 
manipulations. A plurality of computers A, B, . . .N are connected to a 

1 communications network 41, which may comprise an intranet, Ethernet, internet, 
wireless data network, or any equivalent facility that enables signal transmission 
and exchange between the computers. Each computer is provided with software 

;;i0 that includes the DRT function as described herein, as well as the arrow logic 
capability, character recognition capability, pen-based graphic text editing, and 

.!;][ related elements described in the preceding related applications referenced above. 

; B Any two or more computers connected to the network 41 may be linked 

Py for real-time interchange of elements handled by the DRT system. For example, 
15 computers A and N may be linked by a selective handshake routine implemented 
by the respective computer users. Thereafter, for example, the user of computer A 
may draw a DRT diagonal to call forth an array of faders, as indicated by the blue 
dashed line in Figure 21. At the same time, the same fader array with the same 
outer dimensions also appears on the screen of computer N. It is significant to 

20 note that the fader array displayed on computer A are active elements that may be 
manipulated to change levels, arrange or rearrange inputs, and the like. Likewise, 
the fader array communicated to computer N is as fully operational as the array 
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drawn on computer A. Thus active elements may be conveyed directly between 

these machines using the DRT. 

As another example, the steps shown in Figures 19 and 20 to construct a 

letter or similar communication may be transmitted between two or more 
5 computers on the network 41, each DRT stroke being transmitted as it is carried 

out. The recipient computer(s) are enabled to transmit onscreen DRT strokes and 

the like as a reply to any computer from which a DRT element has been received. 
□ For example, as shown in Figure 22, the user of computer N may use a DRT tool 
! f\ to call forth a text portion 42 that has been stored in a window 43, so that the 
R0 text may be scrolled to display it entirely. This text portion 42 and window 32 

'"■'-4 

y will appear at the same time on the display of computer A. Each text portion on 
each computer may be scrolled separately. The ability to share files between 

!!* computers enables truly interactive communications among the computer users, in 

::™ that any user may contribute elements to the onscreen display, and may see the 
15 results of the contributions of other users. 

As a further example, shown in Figure 23, the user of computer N may use 
a DRT stroke to recall a three channel audio processor, an active element that can 
be directed to receive audio inputs and produce individually processed audio 
outputs. This entire audio processor comprises one DRT element that is 

20 transmitted to computer A; after transmission and reception, computer A uses 

software code to generate the active device(s) that the (newly received) onscreen 
graphic elements represent. Computer A may then operate the three channel 
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audio processor independently of the computer N; that is, the audio processor 
now resides independently in computer A and N. 

Using a link (i.e., a network) between two or more computers enables a 
user to use his DRT function to "deliver" fully operational devices, controls, 
5 systems, graphic and text presentations and the like to others on the network by 
simply drawing a diagonal. Furthermore, very complicated consoles or test 
equipment and the like can be constructed at one location and then assigned to a 
DRT. The construction of such consoles, etc. can utilize recognized objects as 
!.n described in patent application serial no. 09/785,049, filed Feb. 15, 2001 entitled: 
Q0 Method For Creating And Operating Control Systems. Then by drawing a DRT 
-1 line onscreen when linked to other computers, this "equipment" can be 

conveyed to anyone on the network. The equipment is delivered as graphic 
information which presents various operations, assignments, etc. Once received, 
;:r{ this graphic assignment information is converted into a complex piece of 
15 equipment.. This operation enables sending complex equipment while utilizing 
very little bandwidth. 

This system is not limited to DRT elements, but may include arrow logic 
transactions and the like. Thus the computers linked through network 41 are 
empowered to conduct fully interactive exchanges in real time. 
20 To summarize, the primary advantages of the DRT system are: 

Increased speed of operation; 

Total customization of the onscreen environment and of the user's 
operational protocols; 
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The substantial elimination of traditional pull down menus, margins, tabs, 
and other time consuming and difficult-to-control setup procedures and 
requirements; 

Greatly improved operation and control over the creation of mixed media, 
5 i.e., pictures, video and audio with text, customized devices, etc.; 

The elimination of required orders of operation (other than sequencing 
with DRT lines), required graphic tools, i.e., grab boxes, select tools, etc. (which 
obscure the clarity of the very objects [elements] that are placed onscreen) and all 
;J] kinds of protocols and rules which must be learned in order to create a simple 
RO document with mixed media; 

The ability of multiple users to transmit and share any element, however 
;=2 complex, that is presented or operated onscreen. 

^ The foregoing description of the preferred embodiment of the invention 

jiy has been presented for purposes of illustration and description. It is not intended 
15 to be exhaustive or to limit the invention to the precise form disclosed, and many 
modifications and variations are possible in light of the above teaching without 
deviating from the spirit and the scope of the invention. The embodiment 
described is selected to best explain the principles of the invention and its 
practical application to thereby enable others skilled in the art to best utilize the 
20 invention in various embodiments and with various modifications as suited to the 
particular purpose contemplated. It is intended that the scope of the invention be 
defined by the claims appended hereto. 
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